{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9F21F2vTIFBa"
      },
      "outputs": [],
      "source": [
        "# Copyright 2025 Google LLC\n",
        "#\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DFO6eX1aIFBc"
      },
      "source": [
        "# Deploying GKE HPA Config Recommender\n",
        "\n",
        "<table align=\"left\">\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://colab.research.google.com/github/aburhan/kubernetes-engine-samples/blob/workloadrecommender/notebook.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\"><br> Open in Colab\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https:%2F%2Fraw.githubusercontent.com%2Faburhan%2Fkubernetes-engine-samples%2Fworkloadrecommender%2Fcost-optimization%2Fhpa-config-recommender%2Fdocs%2Fnotebook.ipynb\">\n",
        "      <img width=\"32px\" src=\"https://cloud.google.com/ml-engine/images/colab-enterprise-logo-32px.png\" alt=\"Google Cloud Colab Enterprise logo\"><br> Open in Colab Enterprise\n",
        "    </a>\n",
        "  </td>    \n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://raw.githubusercontent.com/aburhan/kubernetes-engine-samples/refs/heads/workloadrecommender/cost-optimization/hpa-config-recommender/docs/notebook.ipynb\">\n",
        "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\"><br> Open in Workbench\n",
        "    </a>\n",
        "  </td>\n",
        "  <td style=\"text-align: center\">\n",
        "    <a href=\"https://github.com/aburhan/kubernetes-engine-samples/blob/workloadrecommender/cost-optimization/hpa-config-recommender/docs/notebook.ipynb\">\n",
        "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\"><br> View on GitHub\n",
        "    </a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Overview\n",
        "\n",
        "This notebook guides you through building and running the HPA Config Recommender. This tool analyzes historical metric data from your Google Kubernetes Engine (GKE) workloads to provide optimal Horizontal Pod Autoscaler (HPA) settings. These settings include CPU and memory resource requests, minimum and maximum replica counts, and target utilization, all designed to balance cost efficiency and reliability.\n",
        "\n",
        "> **Note:** This solution is currently tested only for Kubernetes\n",
        "> Deployments.\n",
        "\n",
        "- Key Features\n",
        "\n",
        "-   Fetch and aggregate workload CPU and memory metrics from Cloud\n",
        "    Monitoring.\n",
        "-   Calculate workload startup time by considering pod initialization and\n",
        "    cluster autoscaler delays.\n",
        "-   Simulate resource scaling using DMR (Dynamic Minimum Replicas) and DCR\n",
        "    (Dynamic CPU Requests) algorithms.\n",
        "-   Generate resource recommendations for both HPA and VPA.\n",
        "\n",
        "\n",
        "- Required Roles\n",
        "\n",
        "Ensure you have the following Google Cloud roles:\n",
        "\n",
        "-   roles/resourcemanager.projectCreator\n",
        "-   roles/monitoring.viewer\n",
        "-   roles/bigquery.dataOwner\n",
        "-   roles/artifactregistry.creator\n",
        "-   roles/monitoring.admin\n",
        "\n",
        "### Create a new monitoring project\n",
        "\n",
        "For monitoring workloads across multiple projects, it's best to set up a separate\n",
        "monitoring project. Once you've created this project, you'll need to add your\n",
        "other projects to its metrics scope. This allows you to receive consolidated\n",
        "recommendations. Use the following instructions to\n",
        "[add projects to your metrics scope configuration](https://cloud.google.com/monitoring/settings/multiple-projects)"
      ],
      "metadata": {
        "id": "Oz58HN2AIZfU"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5NeGWWU_ZRa"
      },
      "source": [
        "## Get started\n",
        "\n",
        "This lab will wak you though running the workload recommender. Before you start, make sure you've followed all instructions in [Setup](README.md).\n",
        "\n",
        "The [Setup](README.md) creates a python package repository. You will install the python package to install the library and run a sample simulation on a kubernetes workload."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "PROJECT_ID = \"PROJECT_ID\"  # @param {type:\"string\"}\n",
        "REGION = \"us-central1\"  # @param {type:\"string\"}\n",
        "ARTIFACT_REPO = \"hpa-config-recommender-repo\"\n"
      ],
      "metadata": {
        "id": "siMnB7C5IGZb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4tRO_8qUIFBd"
      },
      "outputs": [],
      "source": [
        "\n",
        "! gcloud auth login --no-launch-browser\n",
        "! gcloud config set project {PROJECT_ID}"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Install build tool and build the project\n"
      ],
      "metadata": {
        "id": "a-9HNfmVR7Uc"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O9iW9oWt_YLv"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.auth.transport.requests import Request\n",
        "from google.auth import default\n",
        "\n",
        "# Get the access token for authentication\n",
        "creds, project = default()\n",
        "creds.refresh(Request())\n",
        "token = creds.token\n",
        "\n",
        "# Set the PIP authentication header\n",
        "os.environ[\"PIP_EXTRA_INDEX_URL\"] = f\"https://oauth2accesstoken:{token}@{REGION}-python.pkg.dev/{PROJECT_ID}/{ARTIFACT_REPO}/simple/\"\n",
        "\n",
        "!gcloud artifacts print-settings python \\\n",
        "    --project={PROJECT_ID} \\\n",
        "    --repository={ARTIFACT_REPO} \\\n",
        "    --location={REGION}\n",
        "\n",
        "# Install the built library\n",
        "! pip install hpaconfigrecommender --upgrade --quiet\n",
        "! pip install nest_asyncio --quiet"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Restart runtime (Colab only)\n",
        "\n",
        "To use the newly installed packages, you must restart the runtime on Google Colab."
      ],
      "metadata": {
        "id": "CRk4nreLWPSO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    import IPython\n",
        "\n",
        "    app = IPython.Application.instance()\n",
        "    app.kernel.do_shutdown(True)"
      ],
      "metadata": {
        "id": "sIf4bL0gVno_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Authenticate your notebook environment (Colab only)\n",
        "\n",
        "Authenticate your environment on Google Colab."
      ],
      "metadata": {
        "id": "i9m_-AmaWYE-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "\n",
        "if \"google.colab\" in sys.modules:\n",
        "\n",
        "    from google.colab import auth\n",
        "\n",
        "    auth.authenticate_user()"
      ],
      "metadata": {
        "id": "XhsZgvfjWYcj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Get GKE workload details\n",
        "\n",
        "The workload must:\n",
        "\n",
        "*   be a resource monitored by the monitoring project. To add project to the monitoring project: [Add monitor project](https://cloud.google.com/monitoring/settings/multiple-projects#add-monitored-project)\n",
        "*   a K8s Deployment object\n",
        "*   have at least 14 days of metrics in Cloud Monitoring\n",
        "\n"
      ],
      "metadata": {
        "id": "A_q2hEBgXDpY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Allow asyncio to work in nested environments like Jupyter Notebooks\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()\n",
        "\n",
        "# Define input parameters for workload details\n",
        "PROJECT_ID = 'PROJECT_ID'  #@param {type:\"string\"}\n",
        "LOCATION = 'LOCATION'  #@param {type:\"string\"}\n",
        "CLUSTER_NAME = 'CLUSTER_NAME'  #@param {type:\"string\"}\n",
        "NAMESPACE = 'NAMESPACE'  #@param {type:\"string\"}\n",
        "CONTROLLER_NAME = 'CONTROLLER_NAME'  #@param {type:\"string\"}\n",
        "CONTROLLER_TYPE = 'CONTROLLER_TYPE'  #@param {type:\"string\"}\n",
        "CONTAINER_NAME = 'CONTAINER_NAME'  #@param {type:\"string\"}"
      ],
      "metadata": {
        "id": "q75KZX43Yjzy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Set workload detail information\n",
        "\n",
        "*   Set workload detail information\n",
        "*   Retrieve the time it takes for the pod to move from a `scheduled` to `ready` state. See [pod lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/) for more details\n"
      ],
      "metadata": {
        "id": "hBmFs8m8Yi12"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "# Import configuration settings and user agent information\n",
        "import hpaconfigrecommender.utils as utils\n",
        "\n",
        "# Import function to retrieve aggregated workload time series data\n",
        "from hpaconfigrecommender.read_workload_startuptime import get_workload_startup_time\n",
        "\n",
        "# Configure logging for debugging and execution tracking\n",
        "logger = logging.getLogger(__name__)\n",
        "\n",
        "\n",
        "# Load configuration settings. for more details about Config. See\n",
        "config = utils.config.Config()\n",
        "\n",
        "# Create WorkloadDetails object using the input parameters\n",
        "workload_details = utils.models.WorkloadDetails(\n",
        "    config=config,\n",
        "    project_id=PROJECT_ID,\n",
        "    cluster_name=CLUSTER_NAME,\n",
        "    location=LOCATION,\n",
        "    namespace=NAMESPACE,\n",
        "    controller_name=CONTROLLER_NAME,\n",
        "    controller_type=CONTROLLER_TYPE,\n",
        "    container_name=CONTAINER_NAME\n",
        ")\n",
        "\n",
        "workload_details = get_workload_startup_time(\n",
        "    config,\n",
        "    workload_details)\n",
        "workload_details"
      ],
      "metadata": {
        "id": "z4c9fqXQZ1oB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "> **Note:** In the event you don't have access to Asset Inventory you can set use kubectl event to set the pod startuptime:\n",
        "\n",
        "```\n",
        "kubectl get events --field-selector involvedObject.name=<pod-name> \\\n",
        "    --sort-by='.metadata.creationTimestamp' -o jsonpath='{range.items[*]}{.reason}{\"\\t\"}{.lastTimestamp}{\"\\n\"}{end}' | \\\n",
        "    awk '/Scheduled/{t1=$2} /ContainersReady/{t2=$2} END{if (t1 && t2) { \"date -d@\"t2\" +%s\" - \"date -d@\"t1\" +%s\" | getline diff; print diff\"s\"} else {print \"Timestamps not found\"}}'\n",
        "```\n",
        "\n",
        "Then set the startup time in the workload\n",
        "\n",
        "```\n",
        "workload_details.scheduled_to_ready_seconds = [ENTER SECONDS BETWEEN SCHEDULE AND READY]\n",
        "```"
      ],
      "metadata": {
        "id": "FYnMqvQijE_k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Get workload timeseriesere\n",
        "\n",
        "Get workload timeseries data from [Cloud Monitoring](https://console.cloud.google.com/monitoring)\n",
        "\n"
      ],
      "metadata": {
        "id": "Hq7VaXV_fV12"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ksDrL7bSD97c"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "from datetime import datetime, timedelta\n",
        "\n",
        "# Import function to retrieve aggregated workload time series data\n",
        "from hpaconfigrecommender.read_workload_timeseries import get_workload_agg_timeseries\n",
        "\n",
        "# Set the time from scheduling to readiness for the workload\n",
        "# workload_details.scheduled_to_ready_seconds = 20\n",
        "\n",
        "# Define the analysis time window\n",
        "end_datetime = datetime.now()  # Current timestamp as the end of the analysis window\n",
        "start_datetime = end_datetime - timedelta(days=14)  # Start time set to 1 day before\n",
        "\n",
        "# Fetch the aggregated workload time series data\n",
        "workload_df = get_workload_agg_timeseries(\n",
        "    config,\n",
        "    workload_details,\n",
        "    start_datetime,\n",
        "    end_datetime\n",
        ")\n",
        "\n",
        "# Return the workload DataFrame\n",
        "workload_df\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Generate HPA Plans\n",
        "\n",
        "The HPA configuration planning process analyzes past performance data (time series data) of your workload. This data is used to generate various combinations of resource allocations (CPU and Memory) and scaling limits (minimum and maximum replicas). These combinations are then evaluated in the subsequent simulation steps to determine the most efficient and reliable HPA configuration."
      ],
      "metadata": {
        "id": "hATW-uUglSvR"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "afyBzLmGBV3D"
      },
      "outputs": [],
      "source": [
        "from hpaconfigrecommender.run_workload_simulation_plan import get_simulation_plans\n",
        "# Use the `get_simulation_plans` function to generate scaling recommendations based on historical data.\n",
        "\n",
        "plans,msg = get_simulation_plans(workload_details, workload_df)\n",
        "\n",
        "for plan in plans:\n",
        "    print(plan)\n",
        "\n",
        "print(msg)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Run HPA Configuration Simulations\n",
        "\n",
        "This step simulates the HPA algorithm using the plans generated previously and your workload's historical data. Plans that cause resource underprovisioning (CPU or memory) are discarded. The most reliable and cost-effective HPA configuration is returned in best_option_df (DataFrame) and JSON format, including recommendations for CPU/memory resources, replica counts, and target CPU utilization. All valid simulation results are available in all_dfs for further analysis and configuration tuning."
      ],
      "metadata": {
        "id": "9883Tvkwz2P6"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "t-xujLb2BwBV"
      },
      "outputs": [],
      "source": [
        "from hpaconfigrecommender.run_workload_simulation_run import run_simulation_plans\n",
        "# Run the simulation\n",
        "# Simulate the plans to evaluate performance and resource optimization.\n",
        "\n",
        "best_option_df, recommendation,  reasons, all_dfs = run_simulation_plans(plans, workload_details, workload_df)\n",
        "\n",
        "# Review Results\n",
        "# Review the generated analysis and recommendations.\n",
        "\n",
        "if best_option_df.empty:\n",
        "    print(\"No suitable recommendations found. Summary:\")\n",
        "    print(reasons)\n",
        "else:\n",
        "    print(\"Recommendations Summary:\")\n",
        "    print(recommendation.to_json())"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Store recommendation in BigQuery\n",
        "\n",
        "Store HPA recommendations in BigQuery for dashboards and alerts. You can store the best recommendation or all reliable ones. If no cost-effective HPA configuration is found, a static VPA is recommended. Storing all recommendations allows for analysis and fine-tuning of configuration settings."
      ],
      "metadata": {
        "id": "wO6bsvyFuj3s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "store_all_recommendations_to_bigquery = True #@param {type:\"boolean\"}\n",
        "\n",
        "from hpaconfigrecommender.run_workload_simulation_run import write_to_bigquery\n",
        "\n",
        "# Define BigQuery details\n",
        "DATASET_ID = \"workload_metrics\"\n",
        "TABLE_ID = \"hpa_forecast_results\"\n",
        "\n",
        "# Store all simulations to BigQuery\n",
        "if store_all_recommendations_to_bigquery:\n",
        "  for df in all_dfs:\n",
        "    write_to_bigquery(df, workload_details, DATASET_ID, TABLE_ID)\n",
        "\n",
        "# Store only the simulation which has the best cost savings and ensures reliability\n",
        "else:\n",
        "  write_to_bigquery(best_option_df, workload_details, DATASET_ID, TABLE_ID)"
      ],
      "metadata": {
        "id": "JZOoZUS5uoOh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Display the most cost effective simulation\n",
        "\n",
        "Plot the recommendation that is the most cost effective while ensuring reliability"
      ],
      "metadata": {
        "id": "oJwzGzRnv7NI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot visualizations for recommendation\n",
        "if best_option_df:\n",
        "  best_option_df.plot(title=\"CPU Recommendation vs Avg Usage\", x=\"window_begin\", y=[\"recommended_cpu_request\",\"avg_container_cpu_usage\"])\n",
        "  best_option_df.plot(title=\"Memory Recommendation vs Avg Usage (MiB)\", x=\"window_begin\", y=[\"recommended_mem_request_mi\",\"max_containers_mem_usage_mi\"])\n",
        "\n",
        "  best_option_df.plot(title=\"CPU Sum Usage vs Recommendation\", x=\"window_begin\", y=[\"hpa_forecast_sum_cpu_up_and_running\",\"sum_container_cpu_usage\"])\n",
        "  best_option_df.plot(title=\"Memory Sum Usage vs Recommendation (MiB)\", x=\"window_begin\", y=[\"hpa_forecast_sum_mem_up_and_running\",\"sum_containers_mem_usage_mi\"])"
      ],
      "metadata": {
        "id": "Negeu4PlwM2Y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Next Steps\n",
        "\n",
        "Your HPA recommendations are now in BigQuery and ready for visualization! You can create dashboards using tools like Looker or any other tool that connects to BigQuery. The data is located in your monitoring project: `PROJECT_ID.DATASET_ID.TABLE_ID.`"
      ],
      "metadata": {
        "id": "sZyUEQoQ1aO7"
      }
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}